From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DE8FC43610 for ; Mon, 26 Nov 2018 13:55:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DDFDA20817 for ; Mon, 26 Nov 2018 13:55:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DDFDA20817 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726684AbeK0At2 (ORCPT ); Mon, 26 Nov 2018 19:49:28 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37216 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726201AbeK0At1 (ORCPT ); Mon, 26 Nov 2018 19:49:27 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0ECFD36809; Mon, 26 Nov 2018 13:55:15 +0000 (UTC) Received: from treble.redhat.com (ovpn-121-105.rdu2.redhat.com [10.10.121.105]) by smtp.corp.redhat.com (Postfix) with ESMTP id F41331A918; Mon, 26 Nov 2018 13:55:12 +0000 (UTC) From: Josh Poimboeuf To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, Ard Biesheuvel , Andy Lutomirski , Steven Rostedt , Peter Zijlstra , Ingo Molnar , Thomas Gleixner , Linus Torvalds , Masami Hiramatsu , Jason Baron , Jiri Kosina , David Laight , Borislav Petkov , Julia Cartwright , Jessica Yu , "H. Peter Anvin" Subject: [PATCH v2 3/4] x86/static_call: Add out-of-line static call implementation Date: Mon, 26 Nov 2018 07:54:59 -0600 Message-Id: <00b08f2194e80241decbf206624b6580b9b8855b.1543200841.git.jpoimboe@redhat.com> In-Reply-To: References: X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Mon, 26 Nov 2018 13:55:15 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add the x86 out-of-line static call implementation. For each key, a permanent trampoline is created which is the destination for all static calls for the given key. The trampoline has a direct jump which gets patched by static_call_update() when the destination function changes. Signed-off-by: Josh Poimboeuf --- arch/x86/Kconfig | 1 + arch/x86/include/asm/static_call.h | 28 ++++++++++++++++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/static_call.c | 54 ++++++++++++++++++++++++++++++ include/linux/static_call.h | 2 +- 5 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/static_call.h create mode 100644 arch/x86/kernel/static_call.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b5286ad2a982..a2a10e0ce248 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -189,6 +189,7 @@ config X86 select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR select HAVE_STACK_VALIDATION if X86_64 + select HAVE_STATIC_CALL_OUTLINE select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_UNSTABLE_SCHED_CLOCK diff --git a/arch/x86/include/asm/static_call.h b/arch/x86/include/asm/static_call.h new file mode 100644 index 000000000000..6e9ad5969ec2 --- /dev/null +++ b/arch/x86/include/asm/static_call.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_STATIC_CALL_H +#define _ASM_STATIC_CALL_H + +/* + * Manually construct a 5-byte direct JMP to prevent the assembler from + * optimizing it into a 2-byte JMP. + */ +#define __ARCH_STATIC_CALL_JMP_LABEL(key) ".L" __stringify(key ## _after_jmp) +#define __ARCH_STATIC_CALL_TRAMP_JMP(key, func) \ + ".byte 0xe9 \n" \ + ".long " #func " - " __ARCH_STATIC_CALL_JMP_LABEL(key) "\n" \ + __ARCH_STATIC_CALL_JMP_LABEL(key) ":" + +/* + * This is a permanent trampoline which does a direct jump to the function. + * The direct jump get patched by static_call_update(). + */ +#define ARCH_DEFINE_STATIC_CALL_TRAMP(key, func) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 4 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(key) " \n" \ + ".type " STATIC_CALL_TRAMP_STR(key) ", @function \n" \ + STATIC_CALL_TRAMP_STR(key) ": \n" \ + __ARCH_STATIC_CALL_TRAMP_JMP(key, func) " \n" \ + ".popsection \n") + +#endif /* _ASM_STATIC_CALL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 8824d01c0c35..82acc8a28429 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -62,6 +62,7 @@ obj-y += tsc.o tsc_msr.o io_delay.o rtc.o obj-y += pci-iommu_table.o obj-y += resource.o obj-y += irqflags.o +obj-y += static_call.o obj-y += process.o obj-y += fpu/ diff --git a/arch/x86/kernel/static_call.c b/arch/x86/kernel/static_call.c new file mode 100644 index 000000000000..8026d176f25c --- /dev/null +++ b/arch/x86/kernel/static_call.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include + +#define CALL_INSN_SIZE 5 + +void static_call_bp_handler(void); +void *bp_handler_dest; + +asm(".pushsection .text, \"ax\" \n" + ".globl static_call_bp_handler \n" + ".type static_call_bp_handler, @function \n" + "static_call_bp_handler: \n" + "ANNOTATE_RETPOLINE_SAFE \n" + "jmp *bp_handler_dest \n" + ".popsection \n"); + +void arch_static_call_transform(void *site, void *tramp, void *func) +{ + s32 dest_relative; + unsigned long insn; + unsigned char insn_opcode; + unsigned char opcodes[CALL_INSN_SIZE]; + + insn = (unsigned long)tramp; + + mutex_lock(&text_mutex); + + insn_opcode = *(unsigned char *)insn; + if (insn_opcode != 0xe8 && insn_opcode != 0xe9) { + WARN_ONCE(1, "unexpected static call insn opcode 0x%x at %pS", + insn_opcode, (void *)insn); + goto done; + } + + dest_relative = (long)(func) - (long)(insn + CALL_INSN_SIZE); + + opcodes[0] = insn_opcode; + memcpy(&opcodes[1], &dest_relative, CALL_INSN_SIZE - 1); + + /* Set up the variable for the breakpoint handler: */ + bp_handler_dest = func; + + /* Patch the call site: */ + text_poke_bp((void *)insn, opcodes, CALL_INSN_SIZE, + static_call_bp_handler); + +done: + mutex_unlock(&text_mutex); +} +EXPORT_SYMBOL_GPL(arch_static_call_transform); diff --git a/include/linux/static_call.h b/include/linux/static_call.h index c8d0da1ef6b2..651f4d784377 100644 --- a/include/linux/static_call.h +++ b/include/linux/static_call.h @@ -149,7 +149,7 @@ struct static_call_key { .func = _func, \ .tramp = STATIC_CALL_TRAMP(key), \ }; \ - ARCH_DEFINE_STATIC_CALL_TRAMP(key, func) + ARCH_DEFINE_STATIC_CALL_TRAMP(key, _func) #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args) -- 2.17.2