From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: stable@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
patches@lists.linux.dev, "Erhard F." <erhard_f@mailbox.org>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
"Masami Hiramatsu (Google)" <mhiramat@kernel.org>,
Nathan Chancellor <nathan@kernel.org>
Subject: [PATCH 6.1 32/46] x86/static_call: Add support for Jcc tail-calls
Date: Thu, 23 Feb 2023 14:06:39 +0100 [thread overview]
Message-ID: <20230223130433.082606881@linuxfoundation.org> (raw)
In-Reply-To: <20230223130431.553657459@linuxfoundation.org>
From: Peter Zijlstra <peterz@infradead.org>
commit 923510c88d2b7d947c4217835fd9ca6bd65cc56c upstream.
Clang likes to create conditional tail calls like:
0000000000000350 <amd_pmu_add_event>:
350: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 351: R_X86_64_NONE __fentry__-0x4
355: 48 83 bf 20 01 00 00 00 cmpq $0x0,0x120(%rdi)
35d: 0f 85 00 00 00 00 jne 363 <amd_pmu_add_event+0x13> 35f: R_X86_64_PLT32 __SCT__amd_pmu_branch_add-0x4
363: e9 00 00 00 00 jmp 368 <amd_pmu_add_event+0x18> 364: R_X86_64_PLT32 __x86_return_thunk-0x4
Where 0x35d is a static call site that's turned into a conditional
tail-call using the Jcc class of instructions.
Teach the in-line static call text patching about this.
Notably, since there is no conditional-ret, in that case patch the Jcc
to point at an empty stub function that does the ret -- or the return
thunk when needed.
Reported-by: "Erhard F." <erhard_f@mailbox.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Link: https://lore.kernel.org/r/Y9Kdg9QjHkr9G5b5@hirez.programming.kicks-ass.net
[nathan: Backport to 6.1:
- Use __x86_return_thunk instead of x86_return_thunk for func in
__static_call_transform()
- Remove ASM_FUNC_ALIGN in __static_call_return() asm, as call
depth tracking was merged in 6.2]
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
arch/x86/kernel/static_call.c | 49 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 46 insertions(+), 3 deletions(-)
--- a/arch/x86/kernel/static_call.c
+++ b/arch/x86/kernel/static_call.c
@@ -9,6 +9,7 @@ enum insn_type {
NOP = 1, /* site cond-call */
JMP = 2, /* tramp / site tail-call */
RET = 3, /* tramp / site cond-tail-call */
+ JCC = 4,
};
/*
@@ -25,12 +26,39 @@ static const u8 xor5rax[] = { 0x2e, 0x2e
static const u8 retinsn[] = { RET_INSN_OPCODE, 0xcc, 0xcc, 0xcc, 0xcc };
+static u8 __is_Jcc(u8 *insn) /* Jcc.d32 */
+{
+ u8 ret = 0;
+
+ if (insn[0] == 0x0f) {
+ u8 tmp = insn[1];
+ if ((tmp & 0xf0) == 0x80)
+ ret = tmp;
+ }
+
+ return ret;
+}
+
+extern void __static_call_return(void);
+
+asm (".global __static_call_return\n\t"
+ ".type __static_call_return, @function\n\t"
+ "__static_call_return:\n\t"
+ ANNOTATE_NOENDBR
+ ANNOTATE_RETPOLINE_SAFE
+ "ret; int3\n\t"
+ ".size __static_call_return, . - __static_call_return \n\t");
+
static void __ref __static_call_transform(void *insn, enum insn_type type,
void *func, bool modinit)
{
const void *emulate = NULL;
int size = CALL_INSN_SIZE;
const void *code;
+ u8 op, buf[6];
+
+ if ((type == JMP || type == RET) && (op = __is_Jcc(insn)))
+ type = JCC;
switch (type) {
case CALL:
@@ -56,6 +84,20 @@ static void __ref __static_call_transfor
else
code = &retinsn;
break;
+
+ case JCC:
+ if (!func) {
+ func = __static_call_return;
+ if (cpu_feature_enabled(X86_FEATURE_RETHUNK))
+ func = __x86_return_thunk;
+ }
+
+ buf[0] = 0x0f;
+ __text_gen_insn(buf+1, op, insn+1, func, 5);
+ code = buf;
+ size = 6;
+
+ break;
}
if (memcmp(insn, code, size) == 0)
@@ -67,9 +109,9 @@ static void __ref __static_call_transfor
text_poke_bp(insn, code, size, emulate);
}
-static void __static_call_validate(void *insn, bool tail, bool tramp)
+static void __static_call_validate(u8 *insn, bool tail, bool tramp)
{
- u8 opcode = *(u8 *)insn;
+ u8 opcode = insn[0];
if (tramp && memcmp(insn+5, tramp_ud, 3)) {
pr_err("trampoline signature fail");
@@ -78,7 +120,8 @@ static void __static_call_validate(void
if (tail) {
if (opcode == JMP32_INSN_OPCODE ||
- opcode == RET_INSN_OPCODE)
+ opcode == RET_INSN_OPCODE ||
+ __is_Jcc(insn))
return;
} else {
if (opcode == CALL_INSN_OPCODE ||
next prev parent reply other threads:[~2023-02-23 13:09 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-23 13:06 [PATCH 6.1 00/46] 6.1.14-rc1 review Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 01/46] drm/etnaviv: dont truncate physical page address Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 02/46] wifi: ath11k: fix warning in dma_free_coherent() of memory chunks while recovery Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 03/46] wifi: rtl8xxxu: gen2: Turn on the rate control Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 04/46] drm/edid: Fix minimum bpc supported with DSC1.2 for HDMI sink Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 05/46] clk: mxl: Switch from direct readl/writel based IO to regmap based IO Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 06/46] clk: mxl: Remove redundant spinlocks Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 07/46] clk: mxl: Add option to override gate clks Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 08/46] clk: mxl: Fix a clk entry by adding relevant flags Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 09/46] powerpc: dts: t208x: Mark MAC1 and MAC2 as 10G Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 10/46] clk: mxl: syscon_node_to_regmap() returns error pointers Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 11/46] sched/psi: Stop relying on timer_pending() for poll_work rescheduling Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 12/46] random: always mix cycle counter in add_latent_entropy() Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 13/46] scsi: libsas: Add smp_ata_check_ready_type() Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 14/46] scsi: hisi_sas: Fix SATA devices missing issue during I_T nexus reset Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 15/46] spi: mediatek: Enable irq when pdata is ready Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 16/46] docs: perf: Fix PMU instance name of hisi-pcie-pmu Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 17/46] KVM: x86: Fail emulation during EMULTYPE_SKIP on any exception Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 18/46] KVM: SVM: Skip WRMSR fastpath on VM-Exit if next RIP isnt valid Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 19/46] KVM: VMX: Execute IBPB on emulated VM-exit when guest has IBRS Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 20/46] can: kvaser_usb: hydra: help gcc-13 to figure out cmd_len Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 21/46] powerpc: dts: t208x: Disable 10G on MAC1 and MAC2 Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 22/46] spi: mediatek: Enable irq before the spi registration Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 23/46] drm/i915: Remove __maybe_unused from mtl_info Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 24/46] KVM: x86: fix deadlock for KVM_XEN_EVTCHN_RESET Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 25/46] selftests: kvm: move declaration at the beginning of main() Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 26/46] powerpc/64s/radix: Fix RWX mapping with relocated kernel Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 27/46] nfp: ethtool: support reporting link modes Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 28/46] nfp: ethtool: fix the bug of setting unsupported port speed Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 29/46] uaccess: Add speculation barrier to copy_from_user() Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 30/46] x86/alternatives: Introduce int3_emulate_jcc() Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 31/46] x86/alternatives: Teach text_poke_bp() to patch Jcc.d32 instructions Greg Kroah-Hartman
2023-02-23 13:06 ` Greg Kroah-Hartman [this message]
2023-02-23 13:06 ` [PATCH 6.1 33/46] Bluetooth: btusb: Add more device IDs for WCN6855 Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 34/46] riscv: remove special treatment for the link order of head.o Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 35/46] arm64: " Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 36/46] arch: fix broken BuildID for arm64 and riscv Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 37/46] powerpc/vmlinux.lds: Define RUNTIME_DISCARD_EXIT Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 38/46] powerpc/vmlinux.lds: Dont discard .rela* for relocatable builds Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 39/46] s390: define RUNTIME_DISCARD_EXIT to fix link error with GNU ld < 2.36 Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 40/46] sh: define RUNTIME_DISCARD_EXIT Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 41/46] wifi: mwifiex: Add missing compatible string for SD8787 Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 42/46] audit: update the mailing list in MAINTAINERS Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 43/46] platform/x86/amd/pmf: Add depends on CONFIG_POWER_SUPPLY Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 44/46] platform/x86: nvidia-wmi-ec-backlight: Add force module parameter Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 45/46] ext4: Fix function prototype mismatch for ext4_feat_ktype Greg Kroah-Hartman
2023-02-23 13:06 ` [PATCH 6.1 46/46] randstruct: disable Clang 15 support Greg Kroah-Hartman
2023-02-23 14:33 ` [PATCH 6.1 00/46] 6.1.14-rc1 review Conor Dooley
2023-02-23 14:37 ` Greg Kroah-Hartman
2023-02-23 14:41 ` Conor Dooley
2023-02-23 14:38 ` Naresh Kamboju
2023-02-23 16:31 ` Greg Kroah-Hartman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230223130433.082606881@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=erhard_f@mailbox.org \
--cc=mhiramat@kernel.org \
--cc=mingo@kernel.org \
--cc=nathan@kernel.org \
--cc=patches@lists.linux.dev \
--cc=peterz@infradead.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).